GraphBuilder: A Scalable Graph ETL Framework

Graph abstraction is essential for many applications, from finding a shortest path to executing complex machine learning (ML) algorithms like collaborative filtering. However, constructing graphs from relationships hidden within large unstructured datasets is challenging. Since graph construction is a data-parallel problem, MapReduce is well-suited for this task. We developed GraphBuilder, an open source scalable framework for graph Extract-Transform-Load (ETL), to offload many of the complexities of graph construction, including graph construction, transformation, normalization, and partitioning. GraphBuilder is written in Java, for ease of programming, and it scales using the MapReduce model. In this paper, we describe the motivation for GraphBuilder, its architecture, MapReduce algorithms for graph processing, and a performance evaluation of the framework. Since large graphs should be partitioned over a cluster for storing and processing and partitioning methods have a significant impact on performance, we develop several graph partitioning methods and evaluate their performance.